When Will Information Retrieval Be “Good Enough”?

We describe a user study that examined the relationship between the quality of an Information Retrieval system and the eectiveness of its users in performing a task. The task involves nding answer facets of questions pertaining to a collection of newswire documents over a six month period. We articially created sets of ranked lists at increasing levels of quality by blending the output of a state-of-the-art retrieval system with truth data created by annotators. Subjects performed the task by using these ranked lists to guide their labeling of answer passages in the retrieved articles. We found that as system accuracy improves, subject time on task and error rate decrease, and the rate of nding new correct answers increases. There is a large intermediary region in which the utility dierence is not signicant; our results suggest that there is some threshold of accuracy for this task beyond which user utility improves rapidly, but more experiments are needed to examine the area around that threshold closely